3D Tracking of Hands Interacting with Several Objects

Humans and robots may acquire knowledge by observing demonstrations of object manipulation in scenarios ranging from everyday tasks, such as tieing laces and executing a recipe, to critical operations like surgery, electronics (dis-)assembly, etc. As the corpus of related videos is enlarged and the knowledge extraction becomes mission critical, the automation of the knowledge extraction process becomes increasingly important. Estimating the detailed configuration of hands and objects in 3D space and across time may be of fundamental importance towards achieving high level understanding of such hand-object interactions. Most relevant work in the literature regards the presented context with identifiable limits. There is plenty of work on the problem of tracking a single hand in 3D. For a review the reader is referred to [8]. There are also a few approaches to the problem of 3D tracking of a single hand and a single object [2, 12], two hands [11] and two hands and a single object [1]. However, in the aforementioned demonstrations it is rarely the case that only a single object is being manipulated. On the contrary, usual interaction scenarios involve several objects, with instances even involving concurrent interaction among sizeable sub-groups (e.g. (dis-)assembly, surgery, etc.). Effective handling of the intrinsically large complexity of these cases constitutes a challenging and interesting goal. We propose that the detailed computation of an entire scene’s 3D configuration should be achieved through top-down model-based 3D tracking. Evidently, top-down methods present straightforward generalization paths, as indicated in the transition from the 3D tracking of a single hand based on RGBD input [10] to single hand and single object 3D tracking from multicamera input [9]. Parametric 3D models of hands and objects can either be designed or acquired [5]. The most important aspect of interaction, i.e. dynamics, can also be modelled in parametric forms [5, 7, 9]. To simplify the incurred optimization problems temporal continuity is assumed, which in turn gives rise to tracking. Here, we present two distinct and complementary approaches, which fall within the same computational framework and push 3D tracking beyond the boundary of two hands and an object. One approach emphasizes on optimization and makes it practical for large scene scales by exploiting the structure of the tracking problem. The other approach focuses on modelling and shows that the exploitation of physics as a prior can lower the dimensionality of the tracking problem.